Automatic Evaluation of Extract Summaries Using Fuzzy F-Score Measure
نویسنده
چکیده
This paper describes a fuzzy union based approach for automatically evaluating machine generated extract summaries. The proposed method represents every sentence within a machine generated summary as a fuzzy set. Sentences in the reference summary are assigned membership grades in each of these fuzzy sets using cosine distance measure. Finally Fuzzy union (s-norm operation) is used to compute an F-score measure. Max s-norm and Frank’s s-norm operators are discussed and the advantages of using a modified Frank’s s-norm operator for evaluating summaries is explained. The proposed evaluation method is compared to ROUGE evaluation system with some test cases.
منابع مشابه
Some Examinations of Intrinsic Methods for Summary Evaluation Based on the Text Summarization Challenge (TSC)
Computer-produced summaries have traditionally been evaluated by comparing them with human-produced summaries using the Fmeasure. However, the F-measure is not appropriate when alternative sentences are possible in a human-produced extract. In this paper, we examine some evaluation methods devised to overcome the problem, including utility-based evaluation. By giving scores for moderately impor...
متن کاملA News Summarization System using Fuzzy Graph Based Document Model
This paper describes a news summarization system using the Fuzzy Graph based Document Model. News articles are modelled as fuzzy graphs whose nodes are sentences and edges are weighted by the fuzzy similarity measure between the sentences. The similarity between sentences is in between 0 and 1. Centrality of the graph retrieves important sentences. The proposed system produces summaries by Eige...
متن کاملCentering Theory for Evaluation of Coherence in Computer-Aided Summaries
This paper investigates a new evaluation method for assessing the coherence of computer-aided summaries, justified by the inappropriacy of existing evaluation methods for this task. It develops a metric for Centering Theory (CT), a theory of local coherence and salience, to measure coherence in pairs of extracts and abstracts produced in a computer-aided summarisation environment. 100 news text...
متن کاملA Learning Technique to Determine Criteria for Multiple Document Summarization
In this paper we describe a new method of automatic summarization based on a learning step to identify criteria that maximize the correlation between human summary and peer extract. The proposed method uses a genetic algorithm to produce extracts from a collection of source documents describing the same event. Theses extracts are compared to human summaries using “Rouge measure” in order to ide...
متن کاملA Comparison Of Rankings Produced By Summarization Evaluation Measures
Summary evaluation measures produce a ranking of all possible extract summaries of a document., Recall-based evaluation measures, which depend on costly human-generated ground truth summaries, produce uncorrelated rankings when ground truth is varied. This paper proposes using sentence-rankbased and content-based measures for evaluating extract summaries, and compares these with recallbased eva...
متن کامل